{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第四题：神经网络：三层感知机"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实现内容：\n",
    "1. 实现一个三层感知机\n",
    "2. 对手写数字数据集进行分类\n",
    "3. 绘制损失值变化曲线\n",
    "4. 完成kaggle MNIST手写数字分类任务，根据给定的超参数训练模型，完成表格的填写"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这道题中，我们要实现一个三层感知机\n",
    "\n",
    "<img src=\"https://davidham3.github.io/blog/2018/09/11/logistic-regression/Fig2.png\" ,width=600>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 前向传播\n",
    "\n",
    "我们实现一个最简单的三层感知机，一个输入层，一个隐藏层，一个输出层，隐藏层单元个数为$h$个，输出层有$K$个单元。\n",
    "\n",
    "1. 我们将第一层的输入，定义为$X \\in \\mathbb{R}^{n \\times m}$，n个样本，m个特征。  \n",
    "2. 输入层到隐藏层之间的权重(weight)与偏置(bias)，分别为$W_1 \\in \\mathbb{R}^{m \\times h}$，$b_1 \\in \\mathbb{R}^{1 \\times h}$。  \n",
    "3. 隐藏层到输出层的权重和偏置分为别$W_2 \\in \\mathbb{R}^{h \\times K}$，$b_2 \\in \\mathbb{R}^{1 \\times K}$。\n",
    "\n",
    "隐藏层的激活函数选用ReLU\n",
    "\n",
    "$$\n",
    "\\mathrm{ReLU}(x) = \\max (0, x)\n",
    "$$\n",
    "\n",
    "我们用$H_1$表示第一个隐藏层的输出值，$O$表示输出层的输出值，这样，前向传播即可定义为\n",
    "\n",
    "$$\n",
    "Z = XW_1 + b_1\\\\\n",
    "H_1 = \\mathrm{ReLU}(Z)\\\\\n",
    "O = H_1 W_2 + b_2\n",
    "$$\n",
    "\n",
    "其中，$H_1 \\in \\mathbb{R}^{n \\times h}$，$O \\in \\mathbb{R}^{n \\times K}$。\n",
    "\n",
    "**注意：这里我们其实是做了广播，将$b_1$复制了$n-1$份后拼接成了维数为$n \\times h$的矩阵，同理，$b_2$也做了广播，拼成了$n \\times K$的矩阵。**\n",
    "\n",
    "最后一层的输出，使用softmax函数激活，得到神经网络计算出的各类的概率值：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\hat{y_i} & = \\mathrm{softmax}(O_i)\\\\\n",
    "& = \\frac{\\exp{(O_i)}}{\\sum^{K}_{k=1} \\exp{(O_k)}}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中，$\\hat{y_i}$表示第$i$类的概率值，也就是输出层第$i$个神经元经$\\mathrm{softmax}$激活后的值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 损失函数\n",
    "\n",
    "损失函数使用交叉熵损失函数：\n",
    "$$\\mathrm{cross\\_entropy}(y, \\hat{y}) = -\\sum^{K}_{k=1}y_k \\log{(\\hat{y_k})}$$\n",
    "\n",
    "这样，$n$个样本的平均损失为：\n",
    "$$\n",
    "\\mathrm{loss} = - \\frac{1}{n} \\sum_n \\sum^{K}_{k=1} y_k \\log{(\\hat{y_k})}\n",
    "$$\n",
    "\n",
    "**注意，这里我们的提到的$\\log$均为$\\ln$，在numpy中为**`np.log`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 反向传播\n",
    "\n",
    "我们使用梯度下降训练模型，求解方式就是求出损失函数对参数的偏导数，即参数的梯度，然后将参数减去梯度乘以学习率，进行参数的更新。\n",
    "$$\n",
    "W := W - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial W}\n",
    "$$\n",
    "其中，$\\alpha$是学习率。\n",
    "\n",
    "在这道题中，交叉熵损失函数的求导比较麻烦，我们先求神经网络的输出层的偏导数，写成链式法则的形式：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial O_i} = \\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial O_i}\n",
    "$$\n",
    "\n",
    "首先求解第一项：\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} = - \\frac{1}{n} \\sum_n \\sum^{K}_{k=1} y_k \\frac{1}{\\hat{y_k}}\n",
    "$$\n",
    "\n",
    "然后求解第二项，因为$\\hat{y_k}$的分母是$\\sum_k \\exp{(O_k)}$，里面包含$O_i$，所以每一个$\\hat{y_k}$的分母都包含$O_i$，这就要求反向传播的时候需要考虑这$K$项，将这$K$项的偏导数加在一起。\n",
    "\n",
    "这$K$项分别为：$\\frac{\\exp{(O_1)}}{\\sum_k \\exp{(O_k)}}$，$\\frac{\\exp{(O_2)}}{\\sum_k \\exp{(O_k)}}$，...，$\\frac{\\exp{(O_i)}}{\\sum_k \\exp{(O_k)}}$，...，$\\frac{\\exp{(O_k)}}{\\sum_k \\exp{(O_k)}}$。\n",
    "\n",
    "显然，这里只有分子带有$O_i$的这项与其他的项不同，因为分子和分母同时包含了$O_i$，而其他的项只有分母包含了$O_i$。\n",
    "\n",
    "这就需要在求解$\\frac{\\partial \\hat{y}}{\\partial O_i}$的时候分两种情况讨论\n",
    "1. 分子带$O_i$\n",
    "2. 分子不带$O_i$\n",
    "\n",
    "第一种情况，当分子含有$O_i$时：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial \\hat{y_i}}{\\partial O_i} & = \\frac{\\partial \\hat{y_i}}{\\partial O_i}\\\\\n",
    "& = \\frac{\\exp{(O_i)} (\\sum^{K}_{k=1} \\exp{(O_k)}) - (\\exp{(O_i)})^2 }{(\\sum^{K}_{k=1} \\exp{(O_k)})^2}\\\\\n",
    "& = \\frac{\\exp{(O_i)}}{\\sum^{K}_{k=1} \\exp{(O_k)}} \\frac{\\sum^{K}_{k=1} \\exp{(O_k)} - \\exp{(O_i)}}{\\sum^{K}_{k=1} \\exp{(O_k)}}\\\\\n",
    "& = \\hat{y_i} ( 1 - \\hat{y_i} )\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "第二种情况，当分子不含$O_i$时，我们用$j$表示当前项的下标：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial \\hat{y_j}}{\\partial O_i} & = \\frac{- \\exp{(O_j)} \\exp{(O_i)}}{(\\sum^{K}_{k=1} \\exp{(O_k)})^2}\\\\\n",
    "& = - \\hat{y_j} \\hat{y_i}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "这样，$\\mathrm{loss}$对$O_i$的偏导数即为：\n",
    "$$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial O_i} & = \\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial O_i}\\\\\n",
    "& = (- \\frac{1}{n} \\sum_n \\sum^{K}_{k=1} y_k \\frac{1}{\\hat{y_k}}) \\frac{\\partial \\hat{y}}{\\partial O_i}\\\\\n",
    "& = - \\frac{1}{n} \\sum_n (y_i \\frac{1}{\\hat{y_i}} \\hat{y_i} ( 1 - \\hat{y_i} ) + \\sum^K_{k \\not= i} y_k \\frac{1}{\\hat{y_k}}( - \\hat{y_k} \\hat{y_i}))\\\\\n",
    "& = - \\frac{1}{n} \\sum_n ( y_i - y_i \\hat{y_i} - \\sum^K_{k \\not= i} y_k \\hat{y_i})\\\\\n",
    "& = - \\frac{1}{n} \\sum_n ( y_i  - \\hat{y_i} \\sum^K_{k = 1} y_k )\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "由于我们处理的多类分类任务，一个样本只对应一个标记，所以$\\sum^K_{k = 1} y_k = 1$，上式在这种问题中，即可化简为：\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial O_i} &= - \\frac{1}{n} \\sum_n ( y_i  - \\hat{y_i})\\\\\n",
    "& = \\frac{1}{n} \\sum_n (\\hat{y_i} -  y_i)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "将其写成矩阵表达式：\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial O} &= \\frac{1}{n} (\\hat{y} - y)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "也就是说，我们的损失函数对输出层的$K$个神经单元的偏导数为$\\mathrm{softmax}$激活值减去真值。\n",
    "\n",
    "接下来我们需要求损失函数对参数$W_2$和$b_2$的偏导数\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial loss}{\\partial W_2} & = \\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial O} \\frac{\\partial O}{\\partial W_2}\\\\\n",
    "& = \\frac{\\partial loss}{\\partial O} \\frac{\\partial O}{\\partial W_2}\\\\\n",
    "& = \\frac{1}{n} (\\hat{y} - y) \\frac{\\partial O}{\\partial W_2}\\\\\n",
    "& = \\frac{1}{n} [{H_1}^\\mathrm{T} (\\hat{y} - y)]\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial loss}{\\partial b_2} & = \\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial O} \\frac{\\partial O}{\\partial b_2}\\\\\n",
    "& = \\frac{\\partial loss}{\\partial O} \\frac{\\partial O}{\\partial b_2}\\\\\n",
    "& = \\frac{1}{n} (\\hat{y} - y) \\frac{\\partial O}{\\partial b_2}\\\\\n",
    "& = \\frac{1}{n} \\sum^n_{i=1} (\\hat{y_i} - y_i)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中，$\\frac{\\partial loss}{\\partial W_2} \\in \\mathbb{R}^{h \\times K}$，$\\frac{\\partial loss}{\\partial b_2} \\in \\mathbb{R}^{1 \\times K}$。  \n",
    "**注意，由于$b_2$是被广播成$n \\times K$的矩阵，因此实际上$b_2$对每个样本的损失都有贡献，因此对其求偏导时，要把$n$个样本对它的偏导数加和。**\n",
    "\n",
    "同理，我们可以求得$\\mathrm{loss}$对$W_1$和$b_1$的偏导数：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial loss}{\\partial W_1} & = \\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial O} \\frac{\\partial O}{\\partial H_1} \\frac{\\partial H_1}{\\partial Z} \\frac{\\partial Z}{\\partial W_1}\\\\\n",
    "& = \\frac{\\partial loss}{\\partial O} \\frac{\\partial O}{\\partial H_1} \\frac{\\partial H_1}{\\partial Z} \\frac{\\partial Z}{\\partial W_1}\\\\\n",
    "& = \\frac{1}{n} {X}^\\mathrm{T} [(\\hat{y} - y) {W_2}^\\mathrm{T} \\frac{\\partial H_1}{\\partial Z}]\\\\\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "由于我们使用的是$\\mathrm{ReLU}$激活函数，它的偏导数为：\n",
    "\n",
    "$$\\frac{\\partial \\mathrm{ReLU(x)}}{\\partial x} = \n",
    "\\begin{cases}\n",
    "0 & \\text{if } x < 0\\\\\n",
    "1 & \\text{if } x \\geq 0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "所以上式为：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial loss}{\\partial {W_1}_{ij}} =\n",
    "\\begin{cases}\n",
    "0 & \\text{if } {Z}_{ij} < 0\\\\\n",
    "    \\frac{1}{n} {X}^\\mathrm{T} (\\hat{y} - y) {W_2}^\\mathrm{T} & \\text{if } {Z}_{ij} \\geq 0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "其中，${W_1}_{ij}$表示矩阵$W_1$第$i$行第$j$列的值，${Z}_{ij}$表示矩阵$Z$第$i$行第$j$列的值。  \n",
    "同理：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial loss}{\\partial b_1} & = \\frac{\\partial \\mathrm{loss}}{\\partial \\hat{y}} \\frac{\\partial \\hat{y}}{\\partial O} \\frac{\\partial O}{\\partial H_1} \\frac{\\partial H_1}{\\partial Z} \\frac{\\partial Z}{\\partial b_1}\\\\\n",
    "& = \\frac{\\partial loss}{\\partial O} \\frac{\\partial O}{\\partial H_1} \\frac{\\partial H_1}{\\partial Z} \\frac{\\partial Z}{\\partial b_1}\\\\\n",
    "& = \\frac{1}{n} (\\hat{y} - y) {W_2}^\\mathrm{T} \\frac{\\partial H_1}{\\partial Z}\\\\\n",
    "& = \\begin{cases}\n",
    "0 &\\text{if } {Z}_{ij} < 0\\\\\n",
    "\\frac{1}{n} \\sum_n (\\hat{y} - y) {W_2}^\\mathrm{T} &\\text{if } {Z}_{ij} \\geq 0\n",
    "\\end{cases}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中，$\\frac{\\partial loss}{\\partial W_1} \\in \\mathbb{R}^{m \\times h}$，$\\frac{\\partial loss}{\\partial b_1} \\in \\mathbb{R}^{1 \\times h}$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参数更新\n",
    "\n",
    "求得损失函数对四个参数的偏导数后，我们就可以使用梯度下降进行参数更新：\n",
    "$$\n",
    "W_2 := W_2 - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial W_2}\\\\\n",
    "b_2 := b_2 - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial b_2}\\\\\n",
    "W_1 := W_1 - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial W_1}\\\\\n",
    "b_1 := b_1 - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial b_1}\\\\\n",
    "$$\n",
    "其中，$\\alpha$是学习率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上内容，就是一个三层感知机的前向传播与反向传播过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 导入数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用第一题的手写数字数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from time import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "40%做测试集，60%做训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(load_digits()['data'], load_digits()['target'], test_size = 0.4, random_state = 32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1078, 64), (1078,), (719, 64), (719,))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX.shape, trainY.shape, testX.shape, testY.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用和第一题一样的标准化处理方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "s = StandardScaler()\n",
    "trainX = s.fit_transform(trainX)\n",
    "testX = s.transform(testX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来还要处理输出。  \n",
    "我们的神经网络是针对每个样本，输出其分别属于$K$类的概率，我们要找最大的那个概率，对应的是哪个类。  \n",
    "我们当前的trainY和testY，每个样本都是一个类标，我们需要将其变成one_hot编码，也就是，假设当前样本的类别是3，我们需要把它变成一个长度为10的向量，其中第4个元素为1，其他元素都为0。得到的矩阵分别记为trainY_mat和testY_mat。  \n",
    "这样，模型训练完成后，会针对每个样本输出十个数，分别代表这个样本属于$0,1,...,9$的概率，那我们只要取最大的那个数的下标，就知道模型认为这个样本是哪类了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "trainY_mat = np.zeros((len(trainY), 10))\n",
    "trainY_mat[np.arange(0, len(trainY), 1), trainY] = 1\n",
    "\n",
    "testY_mat = np.zeros((len(testY), 10))\n",
    "testY_mat[np.arange(0, len(testY), 1), testY] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1078, 10), (719, 10))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainY_mat.shape, testY_mat.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 参数初始化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这题和上一题的区别是，我们把参数用dict存起来"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def initialize(h, K):\n",
    "    '''\n",
    "    参数初始化\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    h: int: 隐藏层单元个数\n",
    "    \n",
    "    K: int: 输出层单元个数\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    parameters: dict，参数，键是\"W1\", \"b1\", \"W2\", \"b2\"\n",
    "    \n",
    "    '''\n",
    "    np.random.seed(32)\n",
    "    W_1 = np.random.normal(size = (trainX.shape[1], h)) * 0.01\n",
    "    b_1 = np.zeros((1, h))\n",
    "    \n",
    "    np.random.seed(32)\n",
    "    W_2 = np.random.normal(size = (h, K)) * 0.01\n",
    "    b_2 = np.zeros((1, K))\n",
    "    \n",
    "    parameters = {'W1': W_1, 'b1': b_1, 'W2': W_2, 'b2': b_2}\n",
    "    \n",
    "    return parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 50)\n",
      "(1, 50)\n",
      "(50, 10)\n",
      "(1, 10)\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "print(parameterst['W1'].shape) # (64, 50)\n",
    "print(parameterst['b1'].shape) # (1, 50)\n",
    "print(parameterst['W2'].shape) # (50, 10)\n",
    "print(parameterst['b2'].shape) # (1, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 前向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "完成Z的计算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def linear_combination(X, W, b):\n",
    "    '''\n",
    "    计算Z，Z = XW + b\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray, shape = (n, m)，输入的数据\n",
    "    \n",
    "    W: np.ndarray, shape = (m, h)，权重\n",
    "    \n",
    "    b: np.ndarray, shape = (1, h)，偏置\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    Z: np.ndarray, shape = (n, h)，线性组合后的值\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # Z = XW + b\n",
    "    # YOUR CODE HERE\n",
    "    Z = np.dot(X,W) + b\n",
    "    \n",
    "    return Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1078, 50)\n",
      "-5.273044421225233e-19\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "print(Zt.shape) # (1078, 50)\n",
    "print(Zt.mean()) # -5.27304442123e-19"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\rm ReLU$激活函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def ReLU(X):\n",
    "    '''\n",
    "    ReLU激活函数\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray，待激活的矩阵\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    activations: np.ndarray, 激活后的矩阵\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # YOUR CODE HERE\n",
    "    activations = np.max(0,X)\n",
    "    \n",
    "    return activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-14-d923687b1cc9>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mparameterst\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0minitialize\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m50\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mZt\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlinear_combination\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtrainX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparameterst\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'W1'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparameterst\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'b1'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0mHt\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mReLU\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mZt\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mHt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# 0.0304\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-13-c788061fe20e>\u001b[0m in \u001b[0;36mReLU\u001b[1;34m(X)\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m     \u001b[1;31m# YOUR CODE HERE\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m     \u001b[0mactivations\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmax\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     17\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     18\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mactivations\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "print(Ht.mean()) # 0.0304\n",
    "\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "print(Ot.shape) # (1078, 10)\n",
    "print(Ot.mean()) # 0.0006"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\rm softmax$激活  \n",
    "\n",
    "$$\n",
    "\\mathrm{softmax}(O_i) = \\frac{\\exp{(O_i)}}{\\sum^{K}_{k=1} \\exp{(O_k)}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def my_softmax(O):\n",
    "    '''\n",
    "    softmax激活\n",
    "    '''\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    return "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例1\n",
    "print(my_softmax(np.array([[0.3, 0.3, 0.3]])))  # array([[ 0.33333333,  0.33333333,  0.33333333]])\n",
    "\n",
    "# 测试样例2\n",
    "test1 = np.array([[-1e32, -1e32, -1e32]])\n",
    "test2 = np.array([[1e32, 1e32, 1e32]])\n",
    "print(my_softmax(test1))\n",
    "print(my_softmax(test2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里，其实是有数值计算上的问题的，假设，我们最后的输出有三个数，每个数都特别小，理论上来说，通过$\\rm softmax$激活后，三个值都是$\\frac{1}{3}$。但实际上就不是这样了，实际上会导致分母为0，除法就不能做了。如果每个数都特别大，会导致做指数运算的时候上溢。\n",
    "\n",
    "我们需要用其他的方法来实现$\\rm softmax$。\n",
    "\n",
    "我们将传入$\\rm softmax$的向量，每个元素减去他们中的最大值，即\n",
    "\n",
    "$$\n",
    "\\mathrm{softmax}(O_i) = \\mathrm{softmax}(O_i - \\mathrm{max(O)})\n",
    "$$\n",
    "\n",
    "这个式子是成立的，感兴趣的同学可以证明一下上面的式子。\n",
    "\n",
    "当我们做了这样的变换后，向量$O$中的最大值就变成了0，就不会上溢了，而分母中最少有一项为1，也不会出现下溢导致分母为0的问题了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def softmax(O):\n",
    "    '''\n",
    "    softmax激活函数\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    O: np.ndarray，待激活的矩阵\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    activations: np.ndarray, 激活后的矩阵\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # YOUR CODE HEER\n",
    "    activations = \n",
    "    \n",
    "    return activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "y_pred = softmax(Ot)\n",
    "\n",
    "print(y_pred.shape)  # (1078, 10)\n",
    "print(Ot.mean())     # 0.000600192658464\n",
    "print(y_pred.mean()) # 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来是实现损失函数，交叉熵损失函数：\n",
    "\n",
    "$$\n",
    "\\mathrm{loss} = - \\frac{1}{n} \\sum_n \\sum^{K}_{k=1} y_k \\log{(\\hat{y_k})}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里又会出一个问题，交叉熵损失函数中，我们需要对$\\rm softmax$的激活值取对数，也就是$\\log{\\hat{y}}$，这就要求我们的激活值全都是大于0的数，不能等于0，但是我们实现的$\\rm softmax$在有些时候确实会输出0，比如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "softmax(np.array([[1e32, 0, -1e32]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这就使得在计算loss的时候会出现问题，解决这个问题的方法是$\\rm log \\ softmax$。所谓$\\rm log \\ softmax$，就是将交叉熵中的对数运算与$\\rm softmax$结合起来，避开为0的情况\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\log{\\frac{\\exp{(O_i)}}{\\sum_K \\exp{(O_k)}}} &= \\log{\\frac{\\exp{(O_i - \\mathrm{max}(O))}}{\\sum_K \\exp{(O_k - \\mathrm{max}(O))}}}\\\\\n",
    "&= O_i - \\mathrm{max}(O) - \\log{\\sum_K \\exp{(O_k - \\mathrm{max}(O))}}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "这样我们再计算$\\rm loss$的时候就可以把输出层的输出直接放到$\\rm log \\ softmax$中计算，不用先激活，再取对数了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们先编写`log_softmax`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def log_softmax(x):\n",
    "    '''\n",
    "    log softmax\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    x: np.ndarray，待激活的矩阵\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    log_activations: np.ndarray, 激活后取了对数的矩阵\n",
    "    \n",
    "    '''\n",
    "    # YOUR CODE HERE\n",
    "    log_activations = \n",
    "    \n",
    "    return log_activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "t = log_softmax(Ot)\n",
    "print(t.shape)  # (1078, 10)\n",
    "print(t.mean()) # -2.30259148717"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后编写`cross_entropy_with_softmax`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def cross_entropy_with_softmax(y_true, O):\n",
    "    '''\n",
    "    求解交叉熵损失函数，这里需要使用log softmax，所以参数分别是真值和未经softmax激活的输出值\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true: np.ndarray，shape = (n, K), 真值\n",
    "    \n",
    "    O: np.ndarray, shape = (n, K)，softmax激活前的输出层的输出值\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    loss: float, 平均的交叉熵损失值\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # 平均交叉熵损失\n",
    "    # YOUR CODE HERE\n",
    "    loss = \n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "losst = cross_entropy_with_softmax(trainY_mat, Ot)\n",
    "print(losst.mean()) # 2.30266707958"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**正是因为$\\rm softmax$激活与交叉熵损失会有这样的问题，所以在很多深度学习框架中，交叉熵损失函数就直接带有了激活的功能，所以我们在实现前向传播计算的时候，就不要加$\\rm softmax$激活函数了。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def forward(X, parameters):\n",
    "    '''\n",
    "    前向传播，从输入一直到输出层softmax激活前的值\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray, shape = (n, m)，输入的数据\n",
    "    \n",
    "    parameters: dict，参数\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    O: np.ndarray, shape = (n, K)，softmax激活前的输出层的输出值\n",
    "    \n",
    "    '''\n",
    "    # 输入层到隐藏层\n",
    "    # YOUR CODE HERE\n",
    "    Z = \n",
    "    \n",
    "    # 隐藏层的激活\n",
    "    # YOUR CODE HERE\n",
    "    H = \n",
    "    \n",
    "    # 隐藏层到输出层\n",
    "    # YOUR CODE HERE\n",
    "    O = \n",
    "\n",
    "    return O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "Ot = forward(trainX, parameterst)\n",
    "print(Ot.mean()) # 0.000600192658464"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "先计算梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def compute_gradient(y_true, y_pred, H, Z, X, parameters):\n",
    "    '''\n",
    "    计算梯度\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true: np.ndarray，shape = (n, K), 真值\n",
    "    \n",
    "    y_pred: np.ndarray, shape = (n, K)，softmax激活后的输出层的输出值\n",
    "    \n",
    "    H: np.ndarray, shape = (n, h)，隐藏层激活后的值\n",
    "    \n",
    "    Z: np.ndarray, shape = (n, h), 隐藏层激活前的值\n",
    "    \n",
    "    X: np.ndarray, shape = (n, m)，输入的原始数据\n",
    "    \n",
    "    parameters: dict，参数\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    grads: dict, 梯度\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # 计算W2的梯度\n",
    "    # YOUR CODE HERE\n",
    "    dW2 = \n",
    "    \n",
    "    # 计算b2的梯度\n",
    "    # YOUR CODE HERE\n",
    "    db2 = \n",
    "    \n",
    "    # 计算ReLU的梯度\n",
    "    relu_grad = Z.copy()\n",
    "    relu_grad[relu_grad >= 0] = 1\n",
    "    relu_grad[relu_grad < 0] = 0\n",
    "    \n",
    "    # 计算W1的梯度\n",
    "    # YOUR CODE HERE\n",
    "    dW1 = \n",
    "    \n",
    "    # 计算b1的梯度\n",
    "    # YOUR CODE HERE\n",
    "    db1 = \n",
    "    \n",
    "    grads = {'dW2': dW2, 'db2': db2, 'dW1': dW1, 'db1': db1}\n",
    "    \n",
    "    return grads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "y_predt = softmax(Ot)\n",
    "\n",
    "gradst = compute_gradient(trainY_mat, y_predt, Ht, Zt, trainX, parameterst)\n",
    "\n",
    "print(gradst['dW1'].sum()) # 0.0429186117668\n",
    "print(gradst['db1'].sum()) # -5.05985151857e-05\n",
    "print(gradst['dW2'].sum()) # -2.16840434497e-18\n",
    "print(gradst['db2'].sum()) # -1.34441069388e-17"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "梯度下降，参数更新"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def update(parameters, grads, learning_rate):\n",
    "    '''\n",
    "    参数更新\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    parameters: dict，参数\n",
    "    \n",
    "    grads: dict, 梯度\n",
    "    \n",
    "    learning_rate: float, 学习率\n",
    "    \n",
    "    '''\n",
    "    parameters['W2'] -= learning_rate * grads['dW2']\n",
    "    parameters['b2'] -= learning_rate * grads['db2']\n",
    "    parameters['W1'] -= learning_rate * grads['dW1']\n",
    "    parameters['b1'] -= learning_rate * grads['db1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "反向传播，参数更新"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "print(parameterst['W1'].sum())  # 0.583495454481\n",
    "print(parameterst['b1'].sum())  # 0.0\n",
    "print(parameterst['W2'].sum())  # 0.1888716431\n",
    "print(parameterst['b2'].sum())  # 0.0\n",
    "print()\n",
    "\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "y_predt = softmax(Ot)\n",
    "\n",
    "gradst = compute_gradient(trainY_mat, y_predt, Ht, Zt, trainX, parameterst)\n",
    "update(parameterst, gradst, 0.1)\n",
    "\n",
    "print(parameterst['W1'].sum())  # 0.579203593304\n",
    "print(parameterst['b1'].sum())  # 5.05985151857e-06\n",
    "print(parameterst['W2'].sum())  # 0.1888716431\n",
    "print(parameterst['b2'].sum())  # 1.24683249836e-18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def backward(y_true, y_pred, H, Z, X, parameters, learning_rate):\n",
    "    '''\n",
    "    计算梯度，参数更新\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true: np.ndarray，shape = (n, K), 真值\n",
    "    \n",
    "    y_pred: np.ndarray, shape = (n, K)，softmax激活后的输出层的输出值\n",
    "    \n",
    "    H: np.ndarray, shape = (n, h)，隐藏层激活后的值\n",
    "    \n",
    "    Z: np.ndarray, shape = (n, h), 隐藏层激活前的值\n",
    "    \n",
    "    X: np.ndarray, shape = (n, m)，输入的原始数据\n",
    "    \n",
    "    parameters: dict，参数\n",
    "    \n",
    "    learning_rate: float, 学习率\n",
    "    \n",
    "    '''\n",
    "    # 计算梯度\n",
    "    # YOUR CODE HERE\n",
    "    grads = \n",
    "    \n",
    "    # 更新参数\n",
    "    # YOUR CODE HERE\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "print(parameterst['W1'].sum())  # 0.583495454481\n",
    "print(parameterst['b1'].sum())  # 0.0\n",
    "print(parameterst['W2'].sum())  # 0.1888716431\n",
    "print(parameterst['b2'].sum())  # 0.0\n",
    "print()\n",
    "\n",
    "Zt = linear_combination(trainX, parameterst['W1'], parameterst['b1'])\n",
    "Ht = ReLU(Zt)\n",
    "Ot = linear_combination(Ht, parameterst['W2'], parameterst['b2'])\n",
    "y_predt = softmax(Ot)\n",
    "\n",
    "backward(trainY_mat, y_predt, Ht, Zt, trainX, parameterst, 0.1)\n",
    "\n",
    "print(parameterst['W1'].sum())  # 0.579203593304\n",
    "print(parameterst['b1'].sum())  # 5.05985151857e-06\n",
    "print(parameterst['W2'].sum())  # 0.1888716431\n",
    "print(parameterst['b2'].sum())  # 1.24683249836e-18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def train(trainX, trainY, testX, testY, parameters, epochs, learning_rate = 0.01, verbose = False):\n",
    "    '''\n",
    "    训练\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    Parameters\n",
    "    ----------\n",
    "    trainX: np.ndarray, shape = (n, m), 训练集\n",
    "    \n",
    "    trainY: np.ndarray, shape = (n, K), 训练集标记\n",
    "    \n",
    "    testX: np.ndarray, shape = (n_test, m)，测试集\n",
    "    \n",
    "    testY: np.ndarray, shape = (n_test, K)，测试集的标记\n",
    "    \n",
    "    parameters: dict，参数\n",
    "    \n",
    "    epochs: int, 要迭代的轮数\n",
    "    \n",
    "    learning_rate: float, default 0.01，学习率\n",
    "    \n",
    "    verbose: boolean, default False，是否打印损失值\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    training_loss_list: list(float)，每迭代一次之后，训练集上的损失值\n",
    "    \n",
    "    testing_loss_list: list(float)，每迭代一次之后，测试集上的损失值\n",
    "    \n",
    "    '''\n",
    "    # 存储损失值\n",
    "    training_loss_list = []\n",
    "    testing_loss_list = []\n",
    "    \n",
    "    for i in range(epochs):\n",
    "        \n",
    "        # 这里要计算出Z和H，因为后面反向传播计算梯度的时候需要这两个矩阵\n",
    "        Z = linear_combination(trainX, parameters['W1'], parameters['b1'])\n",
    "        H = ReLU(Z)\n",
    "        train_O = linear_combination(H, parameters['W2'], parameters['b2'])\n",
    "        train_y_pred = softmax(train_O)\n",
    "        training_loss = cross_entropy_with_softmax(trainY, train_O)\n",
    "        \n",
    "        test_O = forward(testX, parameters)\n",
    "        testing_loss = cross_entropy_with_softmax(testY, test_O)\n",
    "        \n",
    "        if verbose == True:\n",
    "            print('epoch %s, training loss:%s'%(i + 1, training_loss))\n",
    "            print('epoch %s, testing loss:%s'%(i + 1, testing_loss))\n",
    "            print()\n",
    "        \n",
    "        training_loss_list.append(training_loss)\n",
    "        testing_loss_list.append(testing_loss)\n",
    "        \n",
    "        backward(trainY, train_y_pred, H, Z, trainX, parameters, learning_rate)\n",
    "    return training_loss_list, testing_loss_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "parameterst = initialize(50, 10)\n",
    "print(parameterst['W1'].sum())  # 0.583495454481\n",
    "print(parameterst['b1'].sum())  # 0.0\n",
    "print(parameterst['W2'].sum())  # 0.1888716431\n",
    "print(parameterst['b2'].sum())  # 0.0\n",
    "print()\n",
    "\n",
    "training_loss_list, testing_loss_list = train(trainX, trainY_mat, testX, testY_mat, parameterst, 1, 0.1, False)\n",
    "\n",
    "print(parameterst['W1'].sum())  # 0.579203593304\n",
    "print(parameterst['b1'].sum())  # 5.05985151857e-06\n",
    "print(parameterst['W2'].sum())  # 0.1888716431\n",
    "print(parameterst['b2'].sum())  # 1.24683249836e-18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 绘制模型损失值变化曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def plot_loss_curve(training_loss_list, testing_loss_list):\n",
    "    '''\n",
    "    绘制损失值变化曲线\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    training_loss_list: list(float)，每迭代一次之后，训练集上的损失值\n",
    "    \n",
    "    testing_loss_list: list(float)，每迭代一次之后，测试集上的损失值\n",
    "    \n",
    "    '''\n",
    "    plt.figure(figsize = (10, 6))\n",
    "    plt.plot(training_loss_list, label = 'training loss')\n",
    "    plt.plot(testing_loss_list, label = 'testing loss')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.ylabel('loss')\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. 预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型训练完后，我们的就可以进行预测了，需要注意的是，我们的神经网络是针对每个样本，输出其分别属于$K$类的概率，我们要找最大的那个概率，对应的是哪个类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def predict(X, parameters):\n",
    "    '''\n",
    "    预测，调用forward函数完成神经网络对输入X的计算，然后完成类别的划分，取每行最大的那个数的下标作为标记\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray, shape = (n, m), 训练集\n",
    "    \n",
    "    parameters: dict，参数\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    prediction: np.ndarray, shape = (n, 1)，预测的标记\n",
    "    \n",
    "    '''\n",
    "    # 用forward函数得到softmax激活前的值\n",
    "    # YOUR CODE HERE\n",
    "    O = \n",
    "    \n",
    "    # 计算softmax激活后的值\n",
    "    # YOUR CODE HERE\n",
    "    y_pred = \n",
    "    \n",
    "    # 取每行最大的元素对应的下标\n",
    "    # YOUR CODE HERE\n",
    "    prediction = \n",
    "    \n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 测试样例\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "parameterst = initialize(50, 10)\n",
    "training_loss_list, testing_loss_list = train(trainX, trainY_mat, testX, testY_mat, parameterst, 1, 0.1, False)\n",
    "\n",
    "predictiont = predict(testX, parameterst)\n",
    "accuracy_score(predictiont, testY)  # 0.15994436717663421"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. 训练一个三层感知机"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "隐藏层单元数设置为50，输出层单元数为10，我们设置学习率为0.03，迭代轮数为1000轮"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "start_time = time()\n",
    "\n",
    "h = 50\n",
    "K = 10\n",
    "parameters = initialize(h, K)\n",
    "training_loss_list, testing_loss_list = train(trainX, trainY_mat, testX, testY_mat, parameters, 1000, 0.03, False)\n",
    "\n",
    "end_time = time()\n",
    "print('training time: %s s'%(end_time - start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算测试集精度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "prediction = predict(testX, parameters)\n",
    "accuracy_score(prediction, testY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制损失值变化曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "plot_loss_curve(training_loss_list, testing_loss_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 更换数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们换一个数据集，使用MNIST手写数字数据集。\n",
    "\n",
    "MNIST是最有名的手写数字数据集之一，主页：http://yann.lecun.com/exdb/mnist/\n",
    "\n",
    "MNIST手写数字数据集有60000个样本组成的训练集，10000个样本组成的测试集，是NIST的子集。数字的尺寸都是归一化后的，且都在图像的中央。可以从上方的主页下载。\n",
    "\n",
    "我们使用的数据集是kaggle手写数字识别比赛中的训练集。数据集一共42000行，785列，其中第1列是标记，第2列到第785列是图像从左上角到右下角的像素值。图像大小为28×28像素，单通道的灰度图像。\n",
    "\n",
    "我们使用的是kaggle提供的MNIST手写数字识别比赛的训练集。这个数据集还是手写数字的图片，只不过像素变成了 $28 \\times 28$，图片的尺寸变大了，而且数据集的样本量也大了。我们取30%为测试集，70%为训练集。训练集样本数有29400个，测试集12600个。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data/kaggle_mnist/mnist_train.csv')\n",
    "X = data.values[:, 1:].astype('float32')\n",
    "Y = data.values[:, 0]\n",
    "\n",
    "trainX, testX, trainY, testY = train_test_split(X, Y, test_size = 0.3, random_state = 32)\n",
    "\n",
    "trainY_mat = np.zeros((len(trainY), 10))\n",
    "trainY_mat[np.arange(0, len(trainY), 1), trainY] = 1\n",
    "\n",
    "testY_mat = np.zeros((len(testY), 10))\n",
    "testY_mat[np.arange(0, len(testY), 1), testY] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "trainX.shape, trainY.shape, trainY_mat.shape, testX.shape, testY.shape, testY_mat.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制训练集前10个图像"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "_, figs = plt.subplots(1, 10, figsize=(8, 4))\n",
    "for f, img, lbl in zip(figs, trainX[:10], trainY[:10]):\n",
    "    f.imshow(img.reshape((28, 28)), cmap = 'gray')\n",
    "    f.set_title(lbl)\n",
    "    f.axes.get_xaxis().set_visible(False)\n",
    "    f.axes.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## test：请你使用kaggle MNIST数据集，根据下表设定各个超参数，计算测试集上的精度，绘制损失值变化曲线，填写下表\n",
    "\n",
    "任务流程：\n",
    "1. 对数据集进行标准化处理\n",
    "2. 设定学习率和迭代轮数进行训练\n",
    "3. 计算测试集精度\n",
    "4. 绘制曲线"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### 双击此处填写\n",
    "\n",
    "精度保留4位小数；训练时间单位为秒，保留两位小数。\n",
    "\n",
    "隐藏层单元数 | 学习率 | 迭代轮数 | 测试集精度 | 训练时间(秒)\n",
    "-|-|-|-|\n",
    "100 | 0.1 | 50 |  | \n",
    "100 | 0.1 | 100 |  | \n",
    "100 | 0.1 | 150 |  | \n",
    "100 | 0.1 | 500 |  | \n",
    "100 | 0.01 | 500 |  | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
